WAKE COUNTY 
PUBLIC SCHOOL SYSTEM ACCOUNTS RTE 


ETF 2015-16 DRA Report No. 16.10 


Effective Teaching Framework (ETF): 
Final Report 2015-16 


Abstract 


The Effective Teaching Framework (ETF) initiative was designed to provide a common language and lens for 
understanding the skills, strategies, and resources needed for teachers and school leaders to create an 
optimal learning environment for all students. ETF training consisted of six full days of training for 
principals, assistant principals (APs), and selected Central Services staff, as well as an abridged version (24 
hours) for Instructional Resource Teachers (IRTs). Beginning and mentor teachers also received some 
training. Training was provided to all cohorts as planned. Generally, the training sessions were well- 
attended and highly-rated by the participants; in 2015-16, 92.7% agreed it helped them become more 
effective in their roles. The trainings were supplemented by an ETF website that provided a wide range of 
online resources. However, the website was underutilized, with only 26.3% of teachers indicating that they 
used the website to improve their teaching. In most cases, implementation of ETF was limited to individuals 
applying their learning to classroom observations and teacher evaluations. Based on focus group interviews 
with APs, barriers to school-wide ETF efforts included: a) confusion about expectations for orchestrating an 
ETF “rollout” at their schools, b) lack of communication and follow-up from Central Services staff, c) 
leadership changes, d) lack of communal training and planning time among the schools’ administrative 
teams, and e) lack of grassroots support for the initiative at the school level. While training quality was high, 
building stronger buy-in before implementation, communicating clearer expectations, and implementing 
structures to support long-lasting roll-outs at the school level are recommended. 
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What is ETF? 


Wake County Public School System (WCPSS) developed, piloted, and delivered the Effective Teaching 
Framework (ETF) as a professional learning initiative over several years. ETF was designed to support 
the work of teachers by defining and providing models of the research-based skills and strategies 
considered foundational to effective teaching. Topics included essential beliefs, establishing a “growth 
mindset,” motivation (e.g., relationships, expectations, classroom climate), curriculum planning, 
instructional strategies, and cultural proficiency. A map of these pedagogical domains is included in 
Appendix A. Participants were introduced to ETF through six initial days of training, and received several 
books and other written materials related to the sessions, with special emphasis given to The Skillful 
Teacher (Saphier, Haley-Speca, & Gower, 2008) and Mindset: The New Psychology of Success (Dweck, 
2006). To bolster ongoing learning at the school level beyond the initial training the district also 
developed the ETF website, which provided numerous resources for teachers related to the training and 
the state teacher evaluation standards. Resources on the site were indexed to reflect the content of 
ETF, as well as the state teacher evaluation instrument otherwise known as the North Carolina Educator 
Evaluation System (NCEES). Expenditures for ETF in 2014-15 were about $687,000, with expenditures in 
2015-16 of $619,588; most of this cost each year was for development of a district ETF website and 
licensing for the Edivate© website. ETF also took a great deal of time for the district staff who served as 
trainers, as well as the trainees who attended. 


What were the goals of ETF? 


The Effective Teaching Framework (ETF) was designed to provide a common language and lens for 
understanding the skills, strategies, and resources needed for teachers and school leaders to create an 
optimal learning environment for all students. The Pathway of Change for ETF, revised in 2015-16, is 
provided in Figure 1. It illustrates intent for administrators to be trained, and then to apply the skills 
learned in observing and goal-setting with individual teachers, as well as to plan and implement school- 
wide training as appropriate. In turn, this was expected to impact teaching practices and student 
learning. The website resources were intended to support administrators and teachers in this effort. A 
more detailed logic model describing specific outcomes and measures over several school years can be 
found in Appendix B. For multiple reasons, the model and timeframes for accomplishment of these 
goals evolved significantly over time. Implementation data suggested that assessment of stated short 
and long-term outcomes would not be appropriate in 2015-16 as planned. Accomplishment of ETF’s 
long-term goals may still be achieved through the district’s current reshaping of its professional 
development infrastructure to better align with WCPSS’ Vision 2020 Strategic Plan. 
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Figure 1 
Pathway of Change 


Effort: Effective Teaching Framework (ETF) 
Definition from Program Staff: The Effective Teaching Framework is designed to provide a common language 


and lens for understanding the skills, strategies, and resources needed for teachers and school leaders to 
create an optimal learning environment for all students. 
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To What Extent was ETF Implemented as Planned? 


Outcomes of the training were monitored through surveys and personal feedback, but measurement of 
implementation after the training in schools was more difficult to secure. Nevertheless, after collecting 
data from multiple sources, it was determined that: 


e The ETF training sessions were delivered to targeted staff as planned. In 2015-16, new 
principals, assistant principals (APs), and selected Central Services staff received 48 hours of 
training. Elementary and middle school Instructional Resource Teachers (IRTs) received an 
abridged version of the training totaling 24 hours of professional development. Other 
miscellaneous staff, such as new teachers, mentor teachers, Academically Gifted (AG) teachers, 
etc., received training in key elements of ETF at their regularly scheduled meetings. 
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e Attendance rates were exceptionally high (nearly 100%) for school-based administrators, but 
less so for the Instructional Resource Teachers (IRTs) and Central Services staff, especially as the 
training progressed throughout the academic year. Program staff reported that the lower 
attendance rate for IRTs was attributed to the fact that many of them worked in year-round 
schools and were "tracked out" on some of the training days, as well as a perceived lack of 
accountability to leadership if they declined to attend. 


e Overall, satisfaction with the training was high, with 92.7% of the participants agreeing that the 
ETF training would help them become more effective in their roles. However, Chi-Square 
analyses showed that satisfaction with the 2015-16 training was significantly lower than 2014- 
15; for example, in 2015-16 only 88.4% of the participants rated the training as either 
"excellent" or "good," versus 98.1% in 2014-15. The most likely explanations for this variance 
were that: a) different classifications of staff members were being trained (mostly APs in 2014- 
15 versus mostly IRTs in 2015-16), b) the IRTs received a condensed version of the training, and 
c) much of the 2015-16 training relied on videotapes of the training from the previous year, as 
opposed to being almost entirely face-to-face, and the shift in delivery format was not popular 
with all participants. 


e Data from various sources indicate that the APs found the training valuable when observing 
teachers, providing feedback, and directing them to additional resources. Yet, in many cases 
there was persistent confusion about how to implement a school-wide "rollout" of the ETF 
initiative. Survey data revealed that approximately two-thirds (70.8%) of WCPSS teachers were 
familiar with ETF; conversely, approximately one-third were not. 


e The ETF website was intended to be a key resource for implementation, yet it remained 
underutilized. Only 37.3% of WCPSS teachers agreed that they had visited the website to gather 
information, and only 26.3% agreed that they used these online resources to improve their 
teaching. 


e Asample of School Improvement Plans (SIPs) were reviewed to see if ETF had been incorporated 
into goals, key processes, action steps, or planned staff development activities. Approximately 
one-half of the plans contained terminology commonly associated with ETF (e.g., "growth 
mindset" or "cultural proficiency"). However, ETF is broad enough to cover virtually all best 
teaching practices supported by research, so it is difficult to identify language that is uniquely 
germane to ETF. 


Results: Short and Long-Term Outcomes 


Due to the significant changes to the program and the district’s recent retooling of their professional 
development infrastructure, evaluation of short and long-term outcomes stated in Figure 1 was not 
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feasible. Any improvements (or lack thereof) on any of those metrics to date, even if they were 
measurable, could not be fairly attributed to ETF due to these midstream changes in the program. 


Conclusions/Recommendations 


The ETF training was, overall, well-received and well-attended. However, issues with implementation 
prevented ETF from becoming the far-reaching, over-arching, and heavily-used framework for 
instruction that was envisioned at its conception. Thus, as reflective practitioners, we should consider 
what can be learned from the ETF initiative that could help direct future professional learning efforts to 
drive district improvement. A more detailed discussion of DRA's recommendations can be found at the 
end of this report, but can be summarized as follows: 


e Make staff expectations clear and transparent from the onset. Many school-based 
administrators were unsure about how to implement a “rollout” at their schools and what 
successful implementation was supposed to look like. Several administrators reported that they 
had delayed developing a school-based implementation plan because they were waiting for 
clarification from Central Services. 


e Provide a step-by-step roadmap for implementation. The trainings and online resources 
associated with ETF were formidable in size and scale, and many administrators found them 
overwhelming. Providing guidance through more manageable "action steps" increases the 
likelihood of sustainable change, albeit at a slower, more incremental pace. 


e Encourage staff buy-in. Many APs claimed they didn’t have enough time to develop an ETF 
rollout for their school, which means, by extension, they believed other work should be given 
higher priority. Some APs complained that district priorities seemed to evolve from one year to 
the next, leading some to express skepticism about the lifespan of the ETF initiative. In short, 
the level of grassroots support required for sustainable change was lacking, and the district may 
wish to brainstorm ways to help overcome this obstacle. 


e Modify the Professional Learning Model. When staff are trained in annual cohorts by position 
(first principals, then APs, then IRTs, etc.) the content of the learning evolves by the time it 
reaches the classroom teachers, and time constraints and scheduling made planning together as 
an administrative team difficult. Furthermore, a number of APs reported principal turnover at 
their schools, and that having new leadership contributed to a lack of continuity when it came to 
moving forward with ETF. An easy solution to this problem is not immediately apparent, but 
different rollout models are worth consideration, especially since WCPSS is moving forward with 
other large-scale training efforts in 2016-17. 


e Provide adequate support and follow-up. One of the most recurrent themes to emerge from 
the focus group interviews was that the APs complained about a lack of follow-through after the 
training sessions were completed. In short, the APs said that they attended the six days of 
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training, but then heard little else about ETF from Central Services until the schools’ 
administrative teams were allocated time to plan together during a summer work session in 
2015. The breakdown in communication increased with the Central Services’ personnel changes 
that occurred during the implementation phase of the initiative. 


e = Intentionally build implementation science standards into all phases of large-scale initiatives, 
including the initial program planning, implementation, and evaluation stages. When large, 
complex initiatives like ETF are “rolled out” without a solid implementation and evaluation plan 
in place, it is much more difficult to determine what is working and what is not. The district has 
expertise in implementation and research designs that can yield timely feedback during the 
early stages of implementation, as well as more definitive effectiveness determinations once 
implementation is complete. Effective communication and collaboration between WCPSS 
departments, Central Services, and schools will strengthen future professional development 
initiatives such that the intended impacts are eventually realized. 


Background 


What was the research underlying ETF strategies? 


ETF was “evidence-based” in terms of the content of the training and the method of delivery. The 
content of the ETF training was largely based on research about effective instructional strategies, with 
extensive source material referenced in the key texts utilized (Saphier, Haley-Speca, & Gower, 2008; 
Lindsey, Robins, & Terrel, 2009; Dean, Hubbell, Pitler, & Stone, 2012; Dweck, 2006). ETF delivery 
methods were also supported by brain research on adult learning, which indicates that effective 
professional learning experiences builds on prior knowledge, shows respect for the learners, avoids 
cognitive overload (by providing short blocks of new information followed by active engagement by 
participants), and allows opportunities for practice and collaboration with others (Terpstra, 2012). 


WCPSS also considered The Standards for Professional Learning (Learningforward, 2016), which are 
supported by research. These standards point to seven areas as critical for effective professional 
learning that increases educator effectiveness and improvements for students: 


1. Learning communities committed to continuous improvement, collective responsibility, and goal 
alignment; 

2. Resources which support the participants and which are prioritized, monitored, and 
coordinated; 

3. Learning designs that integrate theories, research, and models of human learning; 

4. Data of various types to plan, assess, and evaluate professional learning; 

5. Leadership which is skilled in developing capacity, as well as advocating and creating support 
systems for learning; 

6. Outcomes which are aligned with educator performance and curriculum standards; and 

7. Implementation which reflects Implementation Science research and provides sustained 
support. 
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The implementation model for ETF reflected the first three standards from the Standards for 
Professional Learning more fully than the others. Standard 4 was partially addressed, with impact on 
teacher evaluation ratings and student outcomes impossible to assess due to the rollout design. This in 
line with the findings of Guskey (2003), who reports that much of the research on effective training 
focuses primarily on the delivery of the training and not its impact on teachers’ practices and student 
learning. 


For more detail, see Appendix C and sections of the full report which follow. 


This study of ETF provides findings that are descriptive in nature for implementation trends to date (see 
Table 1). Astronger research design with a comparison group was not feasible because ETF was rolled 
out to all WCPSS schools simultaneously. All members of a position classification group participated in 
the training at the same time (i.e., principals first, then assistant principals and central administrators, 
then Instructional Resource Teachers). Key questions addressed by the study were whether the 
centralized training was implemented as planned, and the extent to which the training was in turn 
applied at the school level. 


Table 1 
Study Design and Supported Conclusions 
Research Design | Conclusions that Can be Drawn 
QO Experimental We can conclude that the program or policy caused changes in 
outcomes because the research design used random assignment. 
O Quasi-Experimental We can reasonably conclude that the program or policy caused 
changes in outcomes because an appropriate comparison strategy 
was used. 
MI Descriptive These designs provide outcome data for the program or policy, but 
MI Quantitative differences cannot be attributed directly to it due to lack of a 
M Qualitative comparative control group. 


Sources: List, Sadoff, & Wagner (2011) and What Works Clearinghouse (2014). 


A variety of quantitative and qualitative data sources were collected for this evaluation, including 
registration and attendance rosters for the centralized training sessions, surveys regarding the quality of 
the training, interviews and focus groups with training participants, and internet resource usage data. 
More detailed information regarding the study methodology can be found in Appendix D. 


This document serves as the final report on the ETF initiative, covering implementation up to and 
including the 2015-16 school year (Figure 2). Changes for 2016-17 included not renewing the Edivate© 
contract, moving some of the website content into Canvas, and moving other key content into the new 
Teaching and Learning professional development infrastructure rather than offering ETF as a separate 
training. As such, ETF is not continuing in its original form in 2016-17. 
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Figure 2 
Reporting Timeline for ETF 


Final Report on 
Status Report on 2015-16 for staff 
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2014-15 for staff 2014-15 for staff elalemelinaleis 


leadership 


Implementation — Training Efforts 


Were the intended target groups trained? 


Prior to 2015-16, central and school administrators provided ETF professional 
learning sessions for principals, assistant principals, and selected Central Services 
staff. In 2015-16, a contractor facilitated 48 hours of training for new principals, 
assistant principals, and additional central staff in three separate cohorts. Two 
additional cohorts of elementary and middle school Instructional Resource 
Teachers (IRTs) received an abridged version of ETF, consisting of eight three- 
hour sessions (24 hours). The 2015-16 training sessions utilized videos from 
training in 2014-15 as well as “live” presentation of material and discussion. The 
attendance rate for new principals and assistant principals was close to 100%, 
but the attendance rates for the other cohorts declined as training progressed. 
Finally, central staff provided training to beginning teachers, teachers for 
Academically Gifted students, mentor teachers, and schools (upon request). 


Given the high number of teachers in WCPSS (over 10,000), it was not considered feasible to provide 
this extensive six-day training in-person directly to all teachers. Leadership staff decided instead to 
focus on those who supported teachers. The expectation was that school leadership staff would share 
the information most needed by their staff as a whole in group meetings, that the learning would be 
applied to teacher observations and evaluations, and that individual teacher feedback would 
incorporate key concepts from the training. Based on data from staff focus groups, the expectation for 
sharing key concepts related to school needs was not understood as fully as the expectation to apply the 
learning in observations and evaluations. 


Virtually all school-based administrators, and many Central Services administrators, were trained in ETF 
during 2013-14 and 2014-15. In 2015-16, a contractor provided training for Instructional Resource 
Teachers (IRTs), selected Central Services staff, and new school-based administrators in five separate 
cohorts. Sessions utilized videos from 2014-15 sessions and “live” presentation of material and 
discussion. The Central Services staff and new school-based administrators received six full days of 
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training (48 hours), while the IRTs received an abridged version of the training that had been condensed 
to 24 total hours. Attendance was highest for new principals and assistant principals, and lower for IRTs 
and Central Services staff (see Table 2). Program staff suggested that the attendance was lower for the 
IRTs than other groups because many of them worked at year-round schools that might have been 
“tracked out” during the training days, and that there seemed to be a lack of accountability to 
leadership if they did not attend. 


Onasmaller scale, staff from Academic Programming and Support provided beginning teachers, mentor 
teachers, and Academically Intellectually Gifted (AIG) teachers with exposure to the most important 
elements of the ETF trainings as time permitted at their regularly scheduled meetings. They also 
provided training on use of the ETF website and to schools upon request. 


Table 2 
2015-16 ETF Attendance Summary by Cohort 


Hours of 
Cohort Group n es Trends/Comments from Program Staff 
Training 


New Principals and ; 
1 . Sys 49 48 Attendance was high - Close to 100%. 
Assistant Principals 


2-3 Central Services Staff 109 48 Attendance dropped as training progressed. 


Attendance fluctuated. IRTs at year-round schools were 


4 — Elementary School IRTs 117 24 ; , Ch 
sometimes "tracked out. 


Attendance was consistent. Many IRTs were part-time 
5 Middle School IRTs 35 24 administrators and preferred to attend the training 
sessions with Cohort 1. 
Total 310 


Costs 


Expenditures for ETF in 2014-15 were about $608,875, with expenditures in 2015-16 of $556,087. As 
shown in Table 3, most of this cost each year was for development of a district ETF website and licensing 
for the Edivate© website. Edivate© (formerly known as PD 360©) describes itself as an on-demand 
professional learning resource that creates a highly personalized learning experience for educators who 
want to improve their practice. The ETF website continues to house videotapes of each training session 
for use by school staff, as well as other district resources and Edivate© videos as of December 2016. 
WCPSS funds were not allocated to continue Edivate© in 2016-17. 


ETF was delivered in multiple-day face-to-face sessions for most cohorts. In 2013-14, ETF training 
cohorts included primarily principals; in 2014-15, ETF involved mostly assistant principals and selected 
central administrators; in 2015-16, those trained included new principals, assistant principals, 
Instructional Resource Teachers, and others (e.g., beginning teachers) in modified formats. 
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Table 3 
Monetary Expenditures for ETF, 2014-15 and 2015-16 


Expense Description Costs in 2014-15 Costs in 2015-16 
Supplies $18,088.26 $52,960.81 
Stipends $73,793.95 50.00 
Contractors $22,406.25 $33,268.75 
Edivate© $494,587.50 $469,857.90 
Total $608,875.96 $556,087.46 


Source: Oracle budget printouts from Academics. 


In addition to these monetary costs, ETF has had large opportunity costs since the professional learning 
was all developed and delivered by district staff. More specifically: 


e In 2014-15, the opportunity costs for the effort included the time of the 18 trainers to prepare 
and deliver the training on nine days, and the time of 376 participants to be away from their 
regular work assignments for six full days. Centrally, the program coordinator, her secretary, 
and two additional coordinating teachers spent a considerable amount of time on ETF training 
and website development. The coordinating teachers reviewed, catalogued, and posted 
materials to Edivate©. An Enloe High School teacher and a team of students videotaped the full 
training (six days total). 


e In 2015-16, opportunity costs were lower, with one consultant facilitating sessions to 310 
participants (Table 2) which utilized the videotapes instead of live trainers in many cases. 
Training for other groups was largely provided through the two coordinating teachers assigned 
to the effort, plus one Human Resources staff member. 


While these costs are high in an absolute sense, costs per staff member impacted are also important to 
consider. The training could still potentially influence all 10,225 teachers in WCPSS as well as those 
trained directly. If ETF had impacted all teachers, the cost per teacher would be $114 (based on 2014-15 
and 2015-16 only — costs for the 2013-14 principal training were not obtained). However, the results of 
this study suggest that a lower percentage of teachers were impacted. While we do not have precise 
data on the percentage impacted, if we estimate that if only 15% of teachers are impacted, the price 
jumps to $760 per teacher. As in all initiatives, implementation in schools is critical — sharing what was 
learned, applying it in work with teachers, and using the websites and other resources as necessary to 
justify the expenditures. 
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Was the ETF training useful and/or well-regarded? 


Overall, high percentages of staff members who completed the training 
evaluation survey had positive views in both 2014-15 and 2015-16. However, the 
2015-16 survey results were significantly lower across all items tested than they 
were in 2014-15. For example, 98% agreed sessions were either “excellent” or 
“good" in 2015-16 versus 88% in 2014-15. Slightly lower ratings may be because 
many staff were given an abridged version of the training, and the 2015-16 
sessions relied heavily on videotapes from the 2014-15 training efforts rather 
than being entirely face-to-face. As the trainer noted, “some people didn’t care 
for the new format.” The variation may also reflect the different classifications of 
staff members that attended the training (e.g., mostly APs in 2014-15 versus 
mostly IRTs in 2015-16). 


In general, training in both 2014-15 and 2015-16 was well-received and considered valuable (Jackl, 
2015). For example, over 92% of respondents in both years indicated that the training helped them be 
more effective in their role. However, a chi-square analysis showed that the responses to every 
evaluation survey item in 2015-16 were significantly lower than they were in 2014-15, with the 
percentage of positive responses dropping between 3.1 percentage points (the training built upon my 
previous knowledge of teaching practices) to 12.3 percentage points (are central services’ expectations 
clear to you). Two of the largest differences were on the question of whether training was excellent or 
good (98% in 2014-15 versus 88% in 2015-16) and whether expectations were clear (91% vs. 78%). 


Three differences between the 2014-15 and 2015-16 training efforts that may have contributed to the 
decline in ratings can be summarized as follows: 


e the 2014-15 training was conducted almost exclusively by live facilitators, whereas the 2015-16 
trainings relied more on video recordings from topics that had been presented the year before; 

e in 2014-15 and 2015-16, the cohorts varied in staff composition (mostly school-based 
administrators in 2014-15 versus mostly IRTs in 2015-16); and 

e the IRTs, who comprised the bulk of the 2015-16 cohort, received an abridged and condensed 
version of the training that may have impacted their overall perception. 


While the modest decline in training ratings is statistically significant across the board, it is still 
important to underscore the fact that an overwhelming majority of participants found the training to be 
useful. Table 4 reports these data in greater detail. 
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Table 4 
ETF Training Rating Comparison — 2014-15 through 2015-16 


2014-15 2015-16 
ETF Training Survey Item 


SA/A SA/A 
% n % 


Change 


The training has already helped me to be more effective in my role. 


The training built on my previous knowledge of teaching practices. 


The training has provided me with more strategies to suggest to 
teachers or other school staff to improve their effectiveness. 


The training has helped or will help me notice areas for improvement 
as | observe teachers. 


Overall, | believe the quality of the training was "excellent" or "good." 


Are central services expectations for application of our learning at the 
ETF sessions clear to you? (Percentage answering "yes.") 


Do you need additional guidance/assistance from central services 
before implementing ETF at your school? (Percentage answering 


no. 


Note: All percentages were rounded to the nearest tenth of one percentage point. 
Note: A chi-square tests showed differences between 2014-15 and 2015-16 to be statistically significant 
(p values ranged from .0001 to .0265). 
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Implementation — School Level 


Did the ETF training reach WCPSS classroom teachers? 


Approximately 71% of WCPSS teachers agreed that they were familiar 
with the ETF initiative. Approximately 80% agreed that administrators 
had stressed some of the “best practices” commonly associated with ETF. 
When surveyed, responses varied by school level; elementary and middle 
school teachers were the most positive; high schools teachers were the 
least positive. 


Since 2013-14 WCPSS has asked teachers to complete an annual survey that collects data on several 
timely initiatives, as well as other projects such as the WCPSS School Report Cards. In 2015-16, four 
survey items were added to better assess the impact of ETF on classroom teachers (Table 5). 


Overall, approximately 71% of WCPSS teachers agreed that they were familiar with ETF, ranging from 
65.7% of the high school teachers to 73.1% of the middle school teachers. While approximately 80% of 
the teachers agreed that their school’s administrators had stressed ETF concepts such as growth 
mindset and/or cultural proficiency, the ETF initiative is a large enough umbrella to cover virtually every 
best teaching practice currently supported by research. The scope and scale of ETF make it difficult to 
quantify exactly how helpful district staff found the framework. 


ETF was originally intended to impact every teacher in the district, but the data suggest that about one- 
fourth of WCPSS schools did not experience an effective school-wide “rollout” of ETF, or if they did, the 
“branding” or importance of the initiative was not reinforced on an ongoing basis. 
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Table 5 
2015-16 Teachers’ Report of ETF Familiarity and Implementation (Self-Reported) 
rosie Strongl Strongl 
Survey Item n Agreement foe Agree Disagree Gia a 

(SA/A) i 

9 9 0, 
iain farniliar with Waka County's Elementary 4,723 72.4% 10.4% 62.0% 21.0% 6.6% 
‘ ‘ Middle 1,887 73.1% 10.2% 62.9% 20.5% 6.5% 

Effective Teaching Framework (ETF) - 

‘initiative High 2,404 65.7% 7.7% 58.0% 24.9% 9.4% 
: All Teachers* 9,117 70.8% 9.7% 61.1% 21.9% 7.3% 
My school's administrators have stressed Elementary 4,716 82.6% 26.5% 56.1% 12.8% 4.5% 
ETF concepts (e.g., growth mindset, best Middle 1,886 77.3% 22.1% 55.2% 16.4% 6.2% 
teaching practices, cultural proficiency, High 2,383 75.7% 19.3% 56.4% 17.3% 7.0% 
etc.) within the past year. All Teachers* 9,089 79.7% 23.7% 56.0% 14.7% 5.5% 


Source: DRA analysis of 2015-16 WCPSS Teacher Survey data. 

Note: All percentages were rounded to the nearest tenth of one percentage point. 

*Note: Teachers working in schools with a broad range of grade levels (e.g., K-8, 6-12, etc.) were not included in 
the disaggregated analyses of elementary (K-5), middle (6-8), and high (9-12) schools. 


Did WCPSS teachers utilize the ETF online resources? 


The ETF website was not accessed by a high percentage of WCPSS teachers in 
2015-16. When surveyed, only 26% of WCPSS teachers agreed that they used the 
ETF website to improve their teaching. When interviewed, school-based 
administrators expressed concerns that they did not have enough time to 
properly preview the quality of the resources before directing teachers to use 
them, and that the website had become stigmatized as being “for teachers who 
weren't doing very well.” Website usage data indicate the number of users per 
month ranged from 1,769 to over 3,321 (out of over 10,000 teachers). EdivateO 
videos were accessed infrequently and use declined across the year. From August 
1, 2015 to January 28, 2016, 82% of schools had fewer than five users of the site; 
81 schools had two or fewer, and 36 schools had no users at all. 


The ETF website was "soft-launched" in the spring of 2015 for principals and assistant principals, but did 
not have a full content "hard-launch" until August 2015 when it was made available to all of the district’s 
teachers. The purpose of the website was to provide vetted, organized resources for school staff on 
various topics related to instruction for use at the school level, as well as be a resource for 
administrators supervising teachers who needed to grow in particular areas. The website was intended 
to be maintained over time so that it could be a continuing resource for school staff. 


The website catalogued and aligned professional development resources (video media, books, website 
links, etc.) to the NC Teacher Evaluation Process Rubric for Evaluating NC Teachers (NCTEP Rubric) and 
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to the Critical Elements of the Map of Pedagogical Knowledge (Appendix A). The website included 
access to “Edivate@©,” a collection of 2,500 videos featuring master teachers demonstrating skills to 
support teachers wanting to become more effective. 


Teacher Surveys 
When surveyed in 2015-16, approximately one-third (37.3%) of WCPSS teachers reported that they had 
visited the ETF website to gather information, and approximately one-fourth (26.3%) agreed that they 


used the ETF website’s online resources to improve their teaching (Table 6). 


Table 6 
2015-16 Teachers’ Utilization of ETF Online Resources (Self-Reported) 


postive ongly Strongly 
Survey Item School Level n Agreement Aeiee Agree Disagree pieace 
(saa) 8 : 
Elementary 4,763 38.5% 4.3% 34.2% 45.8% 15.8% 
| have visited the ETF website to Middle 1,888 40.2% 4.6% 35.6% 44.8% 15.0% 
gather information. High 2,394 32.7% 3.3% 29.4% 44.7% 22.5% 


All Teachers* 9,099 37.3% 4.1% 33.2% 45.3% 17.4% 


| make use of the ETF website's Elementary 4,684 27.4% 2.0% 25.4% 55.1% 17.4% 
searchable database of resources Middle 1,880 27.3% 2.0% 25.3% 55.4% 17.3% 
(e.g., books, articles, Edivate © High 2,380 23.2% 1.6% 21.6% 51.2% 25.7% 
videos) to improve my teaching. All Teachers* 9,046 26.3% 1.9% 244% 54.1% 19.6% 


Source: DRA analysis of 2015-16 WCPSS Teacher Survey data. 

Note: All percentages were rounded to the nearest tenth of one percentage point. 

*Note: Teachers working in schools with a broad range of grade levels (e.g., K-8, 6-12, etc.) were not included in 
the disaggregated analyses of elementary (K-5), middle (6-8), and high (9-12) schools. 


Google-Analytics© 


To better assess district usage of the ETF website, data from two sources were collected. The first set of 
data, obtained from Google Analytics© on a monthly basis from June 2015 to March 2016, shows usage 
of the website as a whole. Access to the main ETF website does not require a login ID or password, so it 
was not possible to identify school-level usage of the website. 


While Google Analytics© can be a useful tool, and proved to be a valuable data source for this 
evaluation, it must be noted that these data present a slightly-flawed assessment of actual website 
usage. This is because these data are gathered using the Internet Protocol (IP) address of the computer 
accessing the website. Thus, if a single user accesses the website from a work computer, and then visits 
the website again from a home computer, he or she would be recorded as a “new visitor” on the second 
visit. Conversely, if a computer is shared by several users, it is theoretically possible that new users 
would be incorrectly classified as “returning visitors.” Nevertheless, as noted earlier, it remains a useful 
tool for gathering general measures of overall website traffic. 
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In the eight-month period from June 2015 through February 2016, about 18,000 users accessed the ETF 
website, with more than half of these users categorized as “returning visitors” (See Figure 3). The 
number per month ranged from 1,691 to over 3,321. The number of sessions spiked during several 
weeks in August and September 2015 when ETF training was conducted in Central Services, but 
otherwise, activity was fairly flat over the course of the eight-month time period. 


Figure 3 
Number of ETF Website Users and Sessions June 2015 — February 2016 


5,000 
4,500 
4,000 
3,500 
3,000 
2,500 
2,000 
1,500 
1,000 

500 


June July Aug. Sept. Nov. Jan Feb 
——# Sessions} 2,247 2,316 | 4,328 3,954 | 2,347 | 2,041 | 2,586 
——# Users 1,935 | 1,769 | 3,321 | 3,199 | 1,931 | 1,691 | 2,110 


Source: Audience Overview Data from Google Analytics© 
Note: Data could not be obtained for October 2015, and December 2015 was excluded 
because the winter break limited the number of instructional days. 


Edivate© 


The ETF website contains access to “Edivate©” (formerly known as PD 360©), which is a proprietary 
website that requires users to log-in with their WCPSS ID and password. Organizations that subscribe to 
Edivate© can receive “usage summary reports” on a monthly basis. The first report that DRA received 
was a five-month cumulative report covering the time period August 1, 2015 to January 28, 2016. 
Subsequently, monthly reports were received from February 11, 2016 through July 11, 2016. Each 
report lists the number of users at each school who viewed the website during the month, the total 
viewing time in minutes, and the number of programs viewed. 


As Table 7 shows, most teachers in the district did not use the Edivate© website, and usage declined 
from the fall of 2015 to the winter and spring of 2016. From August 2015 to January 2016, 83% of 
schools had fewer than five users of the site; 30 schools had no users, and 81 schools had two or fewer. 
About one-fifth of schools had more than five users of the website, however, average total viewing time 
across all schools was only 25 minutes over the five-month period. From February to July 2016, usage of 
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the website was even more minimal. Only 15 schools had one or more users of the website, and no 
schools had more than five users. 


Table 7 
WCPSS Edivate© Usage from August 2015 — July 2016 


Time | TimeSpan 
January 31, 2016 | July 31, 2016 
oe | of users that have viewed the site 
(including Central Office) a a a a 


Number of schools with users that have viewed site Ss fk 


Number of schools with fewer than five users 150 of 172 (87%) | 172 of 172 (100%) 


Average number of users by schools 
Average total viewing time per school 


Source: Usage data summary provided by Edivate© 


In summary, data from Google Analytics© and Edivate© suggest that the ETF website was not used 
during the 2015-16 school year by most schools and teachers. Usage of the website was highest in 
August and September of 2015, but decreased after September and never again reached beginning-of- 
the-year usage levels. The website continued to attract new users in the fall and winter, but usage 
remained consistently low over time. Usage of the Edivate© website remained very low over the entire 
11-month period, and was not funded in 2016-17. 


Did ETF Impact School Improvement Plans (SIPs)? 


Based ona representative sample of Schoo! Improvement Plans (SIPs), specific 
references to ETF were limited. However, some concepts associated with ETF, 
such as “growth mindset” and “cultural proficiency,” were found in one-half of 
the plans. The two sections that contained the most references to ETF concepts 
were the mission, vision, and values statements, and the summary sheet of 
professional development activities. 
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SIP Review 


WCPSS schools develop School Improvement Plans (SIPs) to assess their needs and set goals, with 
annual updates as appropriate. In order to determine the extent to which ETF was referenced in the 
plans, a representative sample was examined, including the SIPs of four elementary schools, three 
middle schools, and three high schools. 


SIP Composition 


SIPs typically contain five major sections: 


e acomprehensive needs assessment; 

e mission, vision, core beliefs, and values statements; 

e asummary of school goals, key processes, and action steps; 
e asummary of professional development activities; and 

e  anintervention planning matrix. 


While direct references to ETF were limited in most sections of the SIP plans examined, concepts 
associated with ETF, such as “growth mindset” and “cultural proficiency” were found in one-half of the 
plans in the sample review. The two sections that contained the most references to ETF concepts were 
the mission, vision, and values statements, and the summary sheet of professional development 
activities. The size and scope of the ETF initiative encompass so many best teaching practices it is 
difficult to claim that any language in the SIPs is uniquely germane to ETF. 


Sample SIP Statements 


As an example, Leesville Road Middle School’s “Values Statement” references utilization of best 
practices in teaching in the following statement: 


e [Utilize] academic growth as the criteria of recognizing student individuality and teaching 
practices focused on enhancing achievement. 


As a second example, Apex High School’s SIP “Summary Sheet of PD Activities for 2016-17” includes the 
following guidance for teachers setting goals: 


e Asecond PDP goal should outline a specific area of growth as identified through the teacher's 
self-assessment on the NC Teacher Evaluation instrument. This may include, but are not limited 


to the following: 
O engagement strategies; 
O Growth Mindset book study; 
oO technology update; and 
O multiple school alignment for upcoming freshmen. 
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ETF and Implementation Science Standards 


The National Implementation Research Network (NIRN) has delineated frameworks and tools to guide 
implementers of initiatives in terms of elements which help support scale-up of effective practices in 
schools. The District Capacity Assessment (DCA), developed by NIRN (Ward et al., 2015), includes 26 
research-supported criteria for effective implementation. The central coordinator for ETF gathered 
approximately 20 staff from the Central Services and school levels (principals and assistant principals) to 
complete the DCA for the first time in a group meeting in July 2015. 


Since the ETF effort was not designed using this set of guidelines, it is not surprising that many ratings 
were low. The effort received the highest marks for the quality of the training itself, staff selection, and 
utilizing data to inform next steps in implementation. The biggest district areas of need were the 
establishment of district and school implementation teams. Some coaching occurred during 
observations and evaluations of teachers, but the key district resource, the ETF website, was never 
utilized extensively. The DCA was not re-administered in summer of 2016, but the key district and 
school implementation teams were not put in place. While it was hoped that the school leadership 
members who were trained would plan and facilitate implementation together, evidence suggests this 
did not occur on most campuses. 


Discussion and Recommendations 


As was noted earlier, this report complied several data sources (listed in Appendix D) to assess the 
extent of ETF implementation. While the original evaluation plan called for an eventual examination of 
outcome variables for students and teachers (Appendix B), the issues with implementation made such 
analyses unfeasible at the writing of this report. 


ETF will evolve as the district moves forward, most likely through the development of online training 
mini-modules deployed through the Canvas© interface. It is also important to remember that selected 
WCPSS staff, including all principals and assistant principals, received up to 48 hours of ETF training. 
This professional learning was, overall, well-received and well-attended, with the vast majority of 
participants agreeing that the training would help them be more effective in their current roles. That 
being said, ETF did not become the far-reaching, over-arching, and heavily-used framework for 
instruction that was envisioned at its conception. Thus, as reflective practitioners, we should consider 
what can be learned from the ETF initiative that could help direct future professional learning efforts to 
drive district improvement: 


e Make staff expectations clear and transparent from the onset. As noted several times 
throughout this report, the ETF training was considered valuable by the trainees, and nearly all 
agreed that it improved their ability to observe classroom teachers and provide more 
meaningful feedback. Yet, when interviewed in focus groups, administrators were unsure about 
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how to implement a “rollout” at their schools and what successful implementation was 
supposed to look like. Several administrators reported that they had delayed developing a 
school-based implementation plan because they were waiting for clarification from Central 
Services; as one AP commented, “we don’t want to do a bunch of work on this [ETF] because 
later on we might be told that we’re doing it wrong.” Another AP told DRA staff, “we want to 
know what we’re going to be held accountable for doing or not doing.” 


e Provide a step-by-step roadmap for implementation. The two expectations that were most 
clearly conveyed during ETF sessions and that administrators seemed to understand and apply 
were that they utilize the Growth Mindset material at their school, and that they apply ETF 
material to teacher observations and feedback. The trainings and online resources associated 
with ETF were formidable in size and scale, and many administrators found them overwhelming 
(e.g., there were literally thousands of training videos available through the district’s 
subscription to Edivate©). Providing guidance through more manageable "action steps" is more 
likely to lead to sustainable change, albeit at a slower, more incremental pace. 


e Encourage staff buy-in. During focus group interviews some administrators reported that they 
were not aware of the ETF initiative before the trainings began, and by that point they had 
already scheduled staff development activities for their teachers during their early release and 
required workdays for the upcoming academic year. Many APs claimed they didn’t have enough 
time to develop an ETF rollout for their school, which means, by extension, they believed other 
work should be given higher priority. APs also complained of “program overload” in their 
schools because so many other initiatives (e.g., MTSS, SIOP, STEM, etc.) were running 
concurrently, and that district priorities seemed to shift and evolve from one year to the next. 
The end result was that several high school APs expressed skepticism about the lifespan of the 
ETF initiative; the elementary and middle school APs were generally more positive in their 
remarks. In short, however, the level of grassroots support required for sustainable change was 
lacking, and the district may wish to brainstorm ways to help overcome this obstacle. 


e Modify the Professional Learning Model. Admittedly, the sheer number of school personnel in 
WCPSS creates a logistical nightmare for those wishing to provide district-wide staff 
development that touches every single teacher. Yet, when staff are trained in annual cohorts by 
position (first principals, then APs, then IRTs, etc.) the content of the learning evolves as it 
reaches each group and is watered down by the time it reaches the classroom teachers. Time 
constraints and scheduling issues will always remain problematic; for example, many APs 
complained that the ETF training sessions were conducted in different cohorts, which made 
planning together as an administrative team almost impossible since everyone was receiving the 
training at different times. Furthermore, a number of APs reported principal turnover at their 
schools, and that having new leadership contributed to a lack of continuity when it came to 
moving forward with ETF. An easy solution to this problem is not immediately apparent, but 


7 WAKE COUNTY 
Data, Research, and Accountability  ciaiatrila tina 


21 


ETF 2015-16 DRA Report No. 16.10 


different rollout models are worth consideration, especially since WCPSS is moving forward with 
other large-scale training efforts in 2016-17. 


e Provide adequate support and follow-up. One of the most recurrent themes to emerge from 
the focus group interviews was that the APs complained about a lack of follow-through after the 
training sessions were completed. In short, the APs said that they attended the six days of 
training, but then heard little else about ETF from Central Services staff until the schools’ 
administrative teams were allocated time to plan together during a summer work session in 
2015. The breakdown in communication increased with the Central Services' personnel changes 
that occurred during the implementation phase of the initiative. As one skeptical AP asserted in 
2015, “if ETF is going to last, it must become strong enough to survive leadership changes.” 
Leadership changes certainly provide challenges for programs seeking longevity and 
sustainability, but such obstacles have been overcome in the past. The Professional Learning 
Team (PLT) initiative which began in 2010, like most programs, appeared vulnerable during its 
infancy. Yet, because support and follow-up were both consistent and persistent after PLTs 
were first introduced, the teams have continued to meet throughout the tenure of various 
superintendents. Now codified in Board Policy 3610, teachers collaborating within their PLTs 
have become a normal part of WCPSS culture — and not just another new idea to improve 
student outcomes. 


e Intentionally build implementation science standards into all phases of large-scale initiatives, 
including the initial program planning, implementation, and evaluation stages. When large, 
complex initiatives like ETF are “rolled out” without a solid implementation and evaluation plan 
in place, it is much more difficult to determine what is working and what is not. The district has 
expertise in implementation and research designs that can yield timely feedback during the 
early stages of implementation, as well as more definitive effectiveness determinations once 
implementation is complete. The recommendations contained in this report can be 
substantiated by the National Implementation Research Network (NIRN), which has published 
research-based suggestions on how to overcome obstacles to change and make desired 
improvements “stick.” In the case of ETF, a District Capacity Assessment (DCA) tool developed 
by NIRN was useful to program staff in 2014-15 for identifying areas of strength and weakness in 
the implementation stage. Effective communication and collaboration between WCPSS 
departments, Central Services, and schools will strengthen future professional development 
initiatives such that the intended impacts are eventually realized. 
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Appendix A 
ETF Map of Pedagogical Knowledge 


MAP OF PEDAGOGICAL KNOWLEDGE 


Adopted from the Map of Pedagogical Knowledge from The Skil/ful Teacher: Building Your Teaching Skills by Saphier, 
Haley-Speca and Gower, p. 5. 6th edition. Research for Better Teaching. www.RBTeach.com. 
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Strategies/Inputs 


Central Office 
Staff Training 
14-15 


Asst. Principal 
Training 14-15 


Principal 
Training 13-14 


Measures: 


e Post knowledge of 
ETF content 
Satisfaction w/ 
training quality 
Self-efficacy in terms 
of ability to train and 
support teachers 
(survey, focus 


groups) 


DRA Report No. 16.10 


Appendix B: 


Logic Model for Effective Teaching Framework Initiative 


2014-15 Outcomes 


Central staff begin 
to apply learning 
to their roles with 
school staff and 
adminstrators. 


School Admins. 
determine 
school-wide 
needs and begin 
to apply learning 
to staff 
evaluations 


Resources: 


Extensive 
Website 


Books 


Measures: 


e School plan for use of 
ETF content 
Use of web resources 
and feedback on value 
Teacher survey (15- 
16?) 
Administrator focus 
groups (16-17 too?) 


2015-16 Outcomes 


Central staff 
continue to apply 
learning 


Teachers get 
web-training 


School staff share 
appropriate aspects 
of ETF with faculty 
and apply to staff 
evaluations 


Increase in use of 
common language; 
changes in teacher 

knowledge and 
behavior 


Measures: 


e Changes in teacher 
behavior (NCEES, 
surveys, 
observations) 

Use of web resources 
and feedback on 
value 

Beliefs Survey results 
(cultural diversity) 


2016-17 Outcomes 


Improved NCEES 
teacher evaluation 
ratings 


Increased 
student 
engagement 
and learning 


Measures: 


e Improved NCEES 
teacher evaluation 
ratings 
Increased student 
engagement as 
measured by 
surveys 
Possible ETF walk- 
throughs? 


---- p= Indirect effects 


—-» = direct effects 
2017-18 Outcomes 


Improved NCEES 
teacher evaluation 
ratings 


Improved 
teacher 
effectiveness on 
state EVAAS 
standards 


Measures: Improved 
teacher ratings as 
measured by EVAAS 
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Appendix C 
Research Related to ETF 


ETF can be characterized as an “evidence-based” initiative. Research support for ETF includes studies 
supporting the content of the training, as well as studies supporting the method of delivery. 


Content of Training 


A key source of inspiration for the ETF effort was the training model developed and employed in 
Montgomery County, Maryland (personal communication with W. Coplen, Nov. 18, 2014) to develop a 
professional growth system which built teacher and administrator understanding of the teacher 
evaluation system and a common understanding of concepts, strategies, and vocabulary. Courses were 
optional but highly recommended and included 36 hours of instruction. These courses were given very 
favorable ratings by those who attended, but impact of the training on student learning has not been 
studied. This training emphasized content in The Skillful Teacher (Saphier, Haley-Speca, and Gower, 
2008). After piloting the material with a contractor from Montgomery County, WCPSS staff decided to 
create a face-to-face training model for all school and selected central administrators to apply in their 
work which fit district needs more closely. In terms of training content, ETF training was largely based 
on research about effective instructional strategies, with extensive source material referenced in the key 
texts utilized (Saphier, et al., 2008; Lindsey, Robins, & Terrel, 2009; Dean, Hubbell, Pitler, & Stone, 2012; 
Dweck, 2006). 


Method of Delivery 


ETF delivery methods were also supported by brain research on adult learning. Terpstra (2012) 
summarized this research and training practices consistent with brain research. Terpstra indicates that 
effective professional learning experiences build on prior knowledge, shows respect for the learners, 
avoids cognitive overload (by providing short blocks of new information followed by active engagement 
by participants), and allows opportunities for practice and collaboration with others. Modeling, 
opportunities for discovery and creativity, challenging content, multisensory delivery, and individualized 
paths for learning were also cited as helpful. ETF incorporated these areas in their face-to-face training 
in 2014-15, with less discussion built into the abbreviated 2015-16 training. 


The Standards for Professional Learning (Learningforward, 2016), which are supported by research, 
point to the following areas as critical for effective professional learning that increases educator 
effectiveness and improvements for students. 


1. Learning communities committed to continuous improvement, collective responsibility, and goal 
alignment; 

2. Resources which support the participants and which are prioritized, monitored, and 
coordinated; 

3. Learning designs that integrate theories, research, and models of human learning; 

4. Data of various types to plan, assess, and evaluate professional learning; 
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5. Leadership which is skilled in developing capacity, as well as advocating and creating support 
systems for learning; 

6. Outcomes which are aligned with educator performance and curriculum standards; 

7. Implementation which reflects change research and provided sustained support. 


Guskey (2003) cautions that much of the research on effective training does not address the impact on 
teachers’ practices and student learning. Three common characteristics in the 13 lists he reviewed were 
enhancing teacher knowledge, providing sufficient time and resources, and including procedures for 
evaluation, which were all reflected in the ETF effort. He cautions that all three need qualifiers for 
effectiveness because of inconsistent and sometimes contradictory research support for each. The ETF 
study was designed to examine the impact on teacher ratings in the evaluation system (NCEES), but 
limited school implementation and low use of the website thus far make that assessment inappropriate. 


The implementation model for ETF reflected the first three standards from the Standards for 
Professional Learning most fully, with Standard 4 partially addressed: 


e Learning Communities: Initially, trainees were organized for group collaboration and learning, 
with base learning communities of central and school staff at each table, along with variable 
groups for some activities. In 2015-16, some trainees (mostly Instructional Resource Teachers) 
received a condensed version of the training that was not as comprehensive (24 hours versus 36 
hours); 

e Resources: These were provided in the form of books and the website; 

e Learning designs: Integrated theories, research, and models for human learning were utilized; 

e Data: A formal evaluation plan was developed, with data on participants’ reflections on training 


quality and focus group results considered as it was available, and these data were considered 
moving forward. The rollout model made teacher and student outcomes difficult to evaluate. 


The leadership and outcomes standards were generally addressed, but not as fully. In terms of 
leadership, the presenters had a clear command of their material and utilized presentation methods to 
support retention of the learning. However, creating support systems for the learning after the sessions 
was weaker. The staff member coordinating the ETF work took a new position at the end of 2014-15, 
and as a result the effort had less dedicated central support in 2015-16. The website was developed as a 
key support, but a late release, limited training, and other issues hampered its effective use. Outcomes 
of the training were monitored through surveys and personal feedback, but measurement of 
implementation after the training in schools was more difficult to secure. 


Implementation Science (NIRN, n.d.) is a relatively recent field of study which provides guidance to any 
program implementation to ensure that a program’s intended effects reach their intended audience 
within a complex organization. In looking at the implementation of ETF, implementation science 
standards were reflected less fully, which is not surprising given that the developers were not familiar 
with this body of research. Implementation Science focuses on sustaining the effort in schools and 
applying change research was not addressed in the training very explicitly. It was clear from the training 
that administrators were to apply the skills learned as they observed and evaluated teachers, but a 
broader implementation at the schools was not explicit except for the growth mindset material. A 
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session at the district-wide leadership conference in August 2015 sought to clarify the expectations and 
provide administrators a chance to consider implementation plans for their school. Change research 
was reflected only in the sense of learning methods applied, but sharing research about building a 
sustained effort in schools was not addressed. Implementation Science focuses on creating conditions 
that make it more likely that initiatives will be implemented and sustained at the target level (in this 
case at the school level). Two essential components are a central implementation team, as well as 
school implementation teams, to provide long-term coordination and support for the effort. Neither of 
these were in place. In terms of resources, focus groups suggested some use of The Skillful Teacher and 
Growth Mindset books, but the use of the website and overall school implementation levels were low. 
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Appendix D 


Evaluation Methods 


The following appendices provide more detailed information on our study design, including evaluation 


methods, the ETF logic model, and each data source. 


Data ._ | When 
Appendix Methods, Return Rates, Notes 
Source Collected 
Created in 2013- | Modifications were made to the logic model after 
14 key leadership left and plans for ETF in 2016-17 
Logic Model B became unclear. Lack of consistent 
Modified in implementation in schools and use of the website 
winter 2015 also contributed. 
eSchool Solutions© electronic registration data, 
eSchools After all training | provided by the Professional Learning 
Training N/A was completed Department, matched to the corresponding 
Records by cohort attendance sign-in sheets from each training 
session. 
An online survey developed by WCPSS staff 
Contractor : 
Teaining N/A Last day of members from the Academics Department and 
Satisfaction training DRA. ETF training participants completed the 
Survey survey at their final training session. 
WCPSS Teacher N/A March 2016 Return rates varied by school, but were 
Survey approximately 90% across the district. 
Focus Groups N/A Spring 2015 and | aps from elementary, middle, and high school 
Spring 2016 levels; “n” varied by session (see text). 
Interviewed new district leaders of the ETF initiative 
; Late spring, and two coordinating teachers plus the contractor 
Interviews N/A early summer who facilitated basic ETF training. These interviews 
were structured around key questions related to 
implementation and effectiveness of the initiative. 
Website Usage June 2015 Baseline ETF website usage data via Google 
Data N/A through Analytics© and Edivate© provided by the 
February 2016 Communications Department. 
Brief literature review of books utilized in the 
Literature professional learning sessions and online references 
Review N/A May 2016 sources through Google Scholar and other internet 


searches. Also consulted with Director of 
Professional Learning. 
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